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Missile  Defense:  Cost  Estimating  Practices  Have  Improved,  and  Continued  Evaluation 
Will  Determine  Effectiveness 

Since  2002,  the  Missile  Defense  Agency  (MDA)  has  been  charged  with  developing  and  fielding 
the  Ballistic  Missile  Defense  System  (BMDS).  This  highly  complex  group  of  systems  is  made  up 
of  a  diverse  collection  of  land-,  sea-,  and  space-based  assets,  including  sensors,  interceptors, 
and  battle  management  located  around  the  globe,  with  the  purpose  of  detecting,  tracking,  and 
defeating  enemy  ballistic  missiles.  For  over  a  decade,  we  have  annually  reported  on  MDA’s 
progress  and  challenges  in  developing  and  fielding  BMDS  capabilities,  as  well  as  transparency, 
accountability,  and  oversight  issues.  A  list  of  our  reports  can  be  found  in  the  enclosure. 

The  Department  of  Defense  has  spent  approximately  $105  billion  since  2002  on  MDA’s  BMDS 
and  plans  to  spend  an  additional  $37.5  billion  from  fiscal  years  2015  through  2019.  However, 
since  we  began  annual  reporting  on  missile  defense  in  2004,  we  have  been  unable  to  assess 
overall  progress  on  cost  because  the  cost  information  MDA  reported  over  the  years  has  been 
incomplete,  inconsistently  reported,  or  lacked  sufficient  quality  for  review.  In  an  effort  to 
establish  a  means  for  us  to  evaluate  MDA’s  cost,  we  have  made  a  number  of 
recommendations — and  Congress  has  passed  a  number  of  laws — that  are  related  to  the 
establishment  of  baselines  for  the  expected  cost  of  certain  elements  of  the  BMDS  and  to 
reporting  deviations  from  the  baseline  as  MDA’s  programs  progress.  MDA  created  those 
baselines  in  2010,  and  in  201 1  we  found  that  the  cost  estimates  underlying  the  baselines  were 
not  high-quality,  according  to  best  practices  contained  in  the  GAO  Cost  Estimating  and 
Assessment  Guide.''  Further,  we  concluded  that  cost  progress  cannot  be  independently 
evaluated  until  MDA  establishes  baselines  that  are  supported  by  reliable,  high-quality  cost 
estimates. 

The  National  Defense  Authorization  Act  for  Fiscal  Year  2014  mandated  that  MDA  submit  a 
report  to  the  congressional  defense  committees  by  December  26,  2014,  describing,  among 
other  things,  the  actions  it  has  taken  to  improve  the  quality  of  its  cost  estimates,  the  schedule  for 
any  remaining  actions,  and  how  those  actions  compare  to  the  best  practices  and  cost- 
estimation  guidelines  recommended  by  GAO.^  In  addition,  the  agency’s  report  must  also  include 
GAO’s  views  on  its  contents.  This  report  includes  our  assessment  of  MDA’s  October  2014  draft 
report  and  the  extent  to  which  MDA’s  past  and  planned  actions  are  in  line  with  GAO  best 


^GAO,  Missile  Defense:  Actions  Needed  to  Improve  Transparency  and  Accountability,  GAO-1 1-372  (Washington, 
D.C.:  Mar.  24,  201 1 )  and  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and  Managing 
Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 

^National  Defense  Authorization  Act  for  Fiscal  Year  2014,  Pub.  L.  No.  113-66,  §  231(c)  (2013).  The  cost  estimates 
that  are  discussed  in  DOD’s  report  are  those  cost  estimates  that  support  cost  baselines  presented  in  MDA’s  annual 
baseline  report  required  by  10  U.S.C.  §  225.  This  baseline  report,  known  as  MDA’s  BMDS  Accountability  Report, 
presents  the  current  estimate  of  the  BMDS  program  baselines  which  include  resource  (cost)  and  schedule  baselines. 
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practices.  Specifically,  our  assessment  focuses  on  four  key  actions  MDA  has  taken  to  improve 
the  quality  of  its  cost  estimates:  (1 )  releasing  a  June  2012  Cost  Estimating  and  Anaiysis 
Handbook,  (2)  performing  internal  cost  reviews,  (3)  assembling  an  independent  cost 
assessment  team,  and  (4)  receiving  independent  cost  estimates  from  the  Office  of  the  Secretary 
of  Defense  for  Cost  Assessment  and  Program  Evaluation  (CAPE).  This  report  is  being 
published  prior  to  the  release  of  MDA’s  report  so  that  it  can  be  included  as  mandated  in  MDAs 
final  report  to  congressional  defense  committees. 

To  assess  the  actions  MDA  has  taken  to  improve  the  quality  of  its  cost  estimates,  we  focused 
on  four  key  areas  that  form  the  basis  of  MDAs  efforts  to  improve  its  cost  estimates.  First,  we 
evaluated  MDA’s  cost  handbook  by  analyzing  the  content  to  see  how  well  it  compares  to  the 
best  practices  in  GAO’s  guide.  Second,  while  MDA  provided  cost  review  scores  for  all  four  of 
GAO’s  characteristics  of  a  high  quality  estimate,  we  focused  our  assessment  on  the 
comprehensive  characteristic.^  We  examined  MDA’s  comprehensive  scores  for  eleven  cost 
estimates  and  determined  whether  we  concurred  or  did  not  concur  with  MDA’s  rating  based  on 
their  rationale  and  supporting  documentation. We  reviewed  the  comprehensive  characteristic 
because,  according  to  GAO’s  guide,  if  the  cost  estimate  is  not  comprehensive  then  it  cannot 
fully  meet  the  other  best  practice  characteristics.  We  also  met  with  MDA  cost  estimators  who 
performed  these  cost  reviews  to  discuss  our  initial  findings  and  obtain  additional  information 
regarding  the  rationale  for  MDA’s  comprehensiveness  scores.  Third,  we  evaluated  the 
effectiveness  of  MDA’s  independent  cost  assessment  team  by  meeting  with  the  team’s  deputy 
director,  reviewing  its  completed  reports,  and  discussing  the  team’s  processes  and  products. 
Lastly,  we  reviewed  recent  CAPE  independent  cost  assessment  results  for  MDA  cost  estimates 
to  see  how  they  compared  with  MDA’s  cost  estimates  and  held  discussions  with  CAPE  cost 
estimating  experts. 

We  conducted  this  performance  audit  from  October  2014  to  December  2014  in  accordance  with 
generally  accepted  government  auditing  standards.  Those  standards  require  that  we  plan  and 
perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusion  based  on  our  audit  objectives. 

Summary 

MDA  has  taken  positive  steps  to  improve  the  quality  of  its  cost  estimates,  and  continued 
evaluation  will  determine  the  effectiveness  of  these  steps.  In  201 1 ,  we  reviewed  MDA  cost 
estimates  and  found  that  none  of  the  estimates  we  reviewed  substantially  met  the  criteria  for 


®The  GAO  cost  guide  calls  out  four  characteristics  of  a  high-quality  and  reliable  cost  estimate  in  that  they  are 
comprehensive,  well  documented,  accurate,  and  credible.  A  cost  estimate  is  considered  comprehensive  when  it 
accounts  for  all  possible  costs  associated  with  a  project,  details  all  cost-influencing  ground  rules  and  assumptions,  is 
technically  reasonable,  is  structured  in  sufficient  detail  to  ensure  that  costs  are  neither  omitted  nor  double-counted, 
and  the  estimating  teams’  composition  is  commensurate  with  the  assignment. 

"^MDA  provided  its  review  of  1 1  cost  estimates  that  support  nine  programs  or  program  components  presenting 
baselines  in  the  annual  baseline  report.  The  nine  program  or  program  components  in  the  baseline  report  are:  Aegis 
Ashore;  Aegis  SM-3  Block  IB  missiles  and  associated  software;  Aegis  Modernization;  Command,  Control,  Battle 
Management  and  Communications  Spiral  8.2;  Ground-based  Midcourse  Defense;  Targets  and  Countermeasures; 
Terminal  High  Altitude  Area  Defense;  Army  Navy/Transportable  Radar  Surveillance  -  Model  2,  and  Upgraded  Early 
Warning  Radar.  The  agency  provided  two  additional  cost  estimates  that  are  included  in  its  baselined  costs  for  Aegis 
Ashore’s  Facilities  and  Global  Deployment. 
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any  of  the  four  characteristics  called  out  in  GAO’s  cost  guide  to  be  considered  high  quality.® 
MDA  has  since  taken  steps  that  are  in-line  with  GAO  best  practices  including:  releasing  an 
internal  cost  estimating  handbook,  performing  internal  cost  reviews  to  evaluate  the  estimates 
against  the  standards  presented  in  the  handbook,  assembling  an  independent  cost  group  to 
perform  an  additional  independent  review  of  the  quality  of  MDA’s  estimates,  and  receiving 
independent  cost  estimates  from  an  external  organization.  Our  evaluation  of  MDA’s  cost 
handbook  shows  that  it  is  well  aligned  with  the  standards  in  the  GAO  cost  guide.  In  addition, 
GAO  concurred  with  29  out  of  33  MDA  cost  review  scores  for  one  of  the  four  GAO 
characteristics  of  a  high  quality  cost  estimate — comprehensiveness — which  indicates  that  the 
agency  can  effectively  assess  the  comprehensiveness  of  its  estimates.  However,  the 
independent  cost  assessment  team  MDA  assembled  in  January  2013  for  an  additional  review  of 
the  quality  of  its  cost  estimates  is  still  defining  its  processes  and  had  completed  reviews  on  only 
one  program’s  estimate  at  the  time  of  our  review.  Therefore,  it  is  too  early  to  determine  the 
effect  that  group  will  have  on  the  quality  of  future  estimates.  Lastly,  while  independent  cost 
assessments  and  estimates  for  MDA  programs  conducted  by  the  CAPE,  DOD’s  principal 
advisory  body  for  independent  cost  estimates  for  DOD  acquisition  programs,  in  the  past  few 
years  have  provided  some  credibility  for  MDA’s  estimates,  about  half  of  the  program  baselines 
presented  in  the  2014  baseline  report  still  remain  unverified. 

Background 

According  to  the  GAO  Cost  Estimating  and  Assessment  Guide,  the  cost  estimate  is  a  critical 
element  in  any  acquisition  process  to  help  decision  makers  evaluate  resource  requirements  at 
milestones  and  other  important  decision  points.®  Cost  estimates  establish  and  defend  budgets 
and  drive  affordability  analysis.  Specifically,  the  guide  calls  out  four  characteristics  of  a  high- 
quality  and  reliable  cost  estimate  in  that  they  are  comprehensive,  well  documented,  accurate, 
and  credible.  A  cost  estimate  is  considered: 

•  Comprehensive:  when  it  accounts  for  all  possible  costs  associated  with  a  project,  details 
all  cost-influencing  ground  rules  and  assumptions,  is  technically  reasonable,  is 
structured  in  sufficient  detail  to  ensure  that  costs  are  neither  omitted  nor  double-counted, 
and  the  estimating  teams’  composition  is  commensurate  with  the  assignment; 

•  Well-documented:  when  supporting  documentation  for  the  estimate  is  accompanied  by  a 
narrative  explaining  the  process,  sources,  and  methods  used  to  create  the  estimate  and 
contains  the  underlying  data  used  to  develop  the  estimate; 

•  Accurate:  when  the  estimate  is  not  overly  conservative  or  too  optimistic,  and  is  based  on 
an  assessment  of  the  costs  most  likely  to  be  incurred;  and 

•  Credible:  when  the  estimate  has  been  cross-checked  with  independent  cost  estimates, 
the  level  of  confidence  associated  with  the  point  estimate — the  best  guess  at  the  cost 
estimate  given  the  underlying  data — has  been  identified,  and  a  sensitivity  analysis  has 
been  conducted.  During  the  sensitivity  analysis,  the  project  examined  the  effect  of 
changing  one  assumption  related  to  each  project  activity  while  holding  all  other  variables 
constant  in  order  to  identify  which  variable  most  affects  the  cost  estimate. 


®GAO-11-372. 

®GAO-09-3SP. 
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In  201 1 ,  we  reviewed  MDA  cost  estimates  and  found  that  none  of  the  estimates  we  reviewed 
substantially  met  any  of  these  characteristics  to  be  considered  high  quality.^  We  recommended 
that  MDA  take  steps  to  ensure  that  the  cost  estimates  that  underlie  the  program  acquisition 
baselines  are  high  quality,  reliable,  and  documented  to  facilitate  external  review.  In  addition. 
Congress  required  the  MDA  Director,  in  the  Fiscal  Year  2014  National  Defense  Authorization 
Act,  to  submit  to  the  congressional  defense  committees  a  report  outlining  the  plans  of  the 
Director  to  improve  the  quality  of  those  cost  estimates.®  According  to  its  October  2014  draft 
report,  MDA  has  taken  a  number  of  steps  over  the  past  few  years  to  improve  the  quality  of  its 
cost  estimates,  including  releasing  a  cost  estimating  handbook,  performing  internal  cost 
reviews,  assembling  an  independent  cost  assessment  team,  and  receiving  independent  cost 
estimates  from  the  CAPE. 

MDA’s  Internal  Cost  Estimating  Handbook  Aligns  Well  with  the  Standards  Detailed  in 
GAO’S  Cost  Guide 

GAO's  evaluation  of  MDA’s  Cost  Estimating  and  Anaiysis  Handbook  shows  that  it  fully  or 
substantially  meets  all  of  the  standards  outlined  in  the  GAO  Cost  Estimating  and  Assessment 
Guide.  GAO’s  cost  guide  discusses  the  importance  of  establishing  a  disciplined  cost  estimating 
process  as  a  basis  for  the  agency  to  create  credible  cost  estimates.  In  June  2012,  MDA 
released  a  Cost  Estimating  and  Anaiysis  Handboo/c  with  the  primary  purpose  of  standardizing 
the  agency’s  cost  analysis  requirements  and  procedures  for  the  preparation,  documentation, 
and  presentation  of  MDA  cost  estimates.  In  developing  its  handbook,  MDA  incorporated  best 
practices  from  the  National  Aeronautics  and  Space  Administration  and  GAO  as  well  as 
comments  from  the  services. 

GAO’s  cost  guide  identifies  12  steps  as  seen  in  table  1  that,  if  followed  correctly,  should  result  in 
reliable  and  valid  cost  estimates  that  management  can  use  for  making  informed  decisions.  The 
chapters  in  the  GAO  guide  provide  supporting  information  associated  with  each  of  those  steps. 
We  found  that  MDA’s  guide  aligns  well  with  the  GAO  cost  guide  with  eight  steps  fully  meeting 
the  GAO  standards  and  four  steps  substantially  meeting  them.  Table  1  provides  more 
information  on  the  results  of  our  analysis. 

Table  1 :  Summary  Assessment  of  MDA  Handbook  Compared  to  GAO  Best  Practices 


GAO  Steps  of  a  High 
Quality  Cost  Estimating 
Process 

GAO  Assessment  of  MDA’s  Handbook 

1 :  Define  Estimates 

Purpose 

Substantially  Met:  MDA’s  Handbook  states  that  cost  estimates  are  generally  used  to 
support  the  budget  making  process,  but  does  not  discuss  varying  purposes  for  the  six 
types  of  cost  estimates  listed  in  the  Handbook. 

2:  Develop  the  estimating 
plan 

Met 

3:  Define  the  program’s 
characteristics 

Met 

4:  Determine  the 
estimating  structure 

Substantially  Met:  MDA’s  Handbook  discusses  the  importance  of  documents  that  detail 
and  describe  the  standardized  structure  for  planned  work,  but  does  not  mention  the 
importance  of  updating  these  documents. 

5:  Identify  ground  rules 

Substantially  Met:  MDA’s  Handbook  states  that  ground  rules  and  assumptions  should 

^GAO-11 -372. 

®Pub.  L  No.  113-66,  §  231(c). 
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GAO  Steps  of  a  High 
Quality  Cost  Estimating 
Process 

GAO  Assessment  of  MDA’s  Handbook 

and  assumptions 

be  documented  thoroughly,  developed  with  members  of  the  technical  community, 
approved  by  management,  and  tied  directly  to  risk  and  sensitivity  analyses.  However,  it 
does  not  mention  the  relation  of  the  ground  rules  and  assumptions  to  historical  data. 

6:  Obtain  the  data 

Substantially  Met:  MDA’s  Handbook  states  that  historical  data  is  the  foundation  of  a 
credible  cost  estimate  and  discusses  the  normalization  and  analysis  of  data,  data 
collection  plan,  and  ensuring  that  adequate  time  is  allotted  for  data  collection  activities. 
However,  MDA’s  Handbook  states  that  secondary  data  has  advantages  over  primary 
data,  which  is  in  contradiction  to  GAO  best  practices. 

7:  Develop  the  point 
estimate  and  compare  to 
an  independent  cost 
estimate 

Met 

8:  Conduct  a  sensitivity 
Analysis 

Met 

9:  Conduct  a  risk  analysis 

Met 

10:  Document  the 
estimate 

Met 

1 1 :  Present  estimate  to 
management 

Met 

12:  Update  the  estimate 

Met 

Source:  GAO  analysis  of  MDA  data.  |  GAO-1 5-21  OR 

NOTE:  Met:  MDA  provided  complete  evidence  that  satisfies  the  elements  of  the  best  practice;  Substantially  Met: 

MDA  provided  evidence  that  satisfies  a  large  portion  of  the  elements  of  the  best  practice;  Partially  Met:  MDA  provided 
evidence  that  satisfies  about  half  of  the  elements  of  the  best  practice;  Minimally  Met:  MDA  provided  evidence  that 
satisfies  a  small  portion  of  the  elements  of  the  best  practice;  and  Did  Not  Meet:  MDA  provided  no  evidence  that 
satisfies  any  of  the  elements  of  the  best  practice. 

GAO  Review  Indicates  MDA  Can  Effectively  Assess  Comprehensiveness  of  Estimates 

After  examining  MDA  cost  review  scores  for  one  of  the  four  GAO  characteristics  of  a  high 
quality  cost  estimate — comprehensiveness — GAO  concurred  with  29  out  of  33  of  the  scores, 
which  indicates  that  the  agency  can  effectively  assess  the  comprehensiveness  of  its  estimates. 
In  the  past  few  years,  MDA  has  created  an  internal  cost  review  process  to  continuously  evaluate 
the  development  and  documentation  of  its  program  cost  estimates.  Given  that  MDA’s  cost 
handbook  is  well  aligned  with  GAO  standards  and  best  practices,  the  next  step  is  ensuring  that 
the  agency  is  implementing  the  practices  and  procedures  outlined  in  its  handbook  that  will  work 
to  improve  the  quality  of  its  estimates.  GAO’s  cost  guide  states  that  relying  on  a  standard 
process  that  emphasizes  pinning  down  the  technical  scope  of  the  work,  communicating  the 
basis  on  which  the  estimate  is  built,  identifying  the  quality  of  the  data,  determining  the  level  of 
risk,  and  thoroughly  documenting  the  effort  should  result  in  cost  estimates  that  are  defensible, 
consistent,  and  trustworthy.  As  part  of  MDA’s  cost  review  process,  members  of  MDA’s  cost 
estimating  directorate  meet  annually  to  compare  each  of  the  program  estimates  to  the 
standards  in  MDA’s  handbook.  The  meeting  results  in  the  team  assigning  scores  for  each  of  the 
characteristics  associated  with  a  high  quality  estimate  as  outlined  in  MDA’s  handbook.  Possible 
scores  include  “Met”,  “Substantially  Met”,  “Partially  Met”,  “Minimally  Met”,  and  “Not  Met”. 

After  reviewing  MDA’s  scores  for  one  of  the  four  GAO  characteristics  of  a  high  quality  estimate, 
comprehensiveness,  we  determined  that  we  concurred  with  MDA’s  rationale  and  supporting 
documentation  for  29  out  of  33,  or  about  90  percent  of  MDA’s  ratings  as  seen  in  table  2. 
According  to  MDA’s  handbook,  the  agency  scored  the  comprehensiveness  of  the  estimates  by 
averaging  the  scores  for  three  criteria  based  on  how  well  the  estimate: 
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1.  Completely  defined  the  program,  reflected  the  current  schedule,  and  contained 
technically  reasonable  assumptions 

2.  Captured  the  complete  technical  scope  of  the  work  to  be  performed,  used  a  logical  work 
breakdown  structure  that  accounts  for  all  performance  criteria  and  requirements 

3.  Included  a  complete  life  cycle  cost  estimate,  accounting  for  development  (if  any), 
procurement,  operation  and  support  (as  applicable),  AND  disposal 

For  the  MDA  score  for  each  of  the  three  criteria  for  1 1  cost  estimates,  our  evaluation  resulted  in 
either  a  “concur”  or  “does  not  concur”  rating.  There  were  a  few  cases  where  we  could  not 
concur  with  MDA’s  ratings  because  of  some  inconsistencies  among  the  cost  estimate 
documentation  and  areas  where  the  estimate  was  not  complete. 

Table  2:  GAO  Review  of  the  MDA  Comprehensive  Characteristic  Ratings  for  Program  Cost  Estimates 


MDA  Cost 
Estimate 

MDA  Rating  for 
Comprehensive 
Characteristic 

Brief  Explanation  of  GAO  Assessment 

Aegis  Ashore 

Met 

GAO  did  not  concur  with  one  of  the  three  criteria  for  MDA’s  rating 
because  some  of  the  operations  and  sustainment  costs  were  not 
included  in  the  roll-up  estimating  model. 

Aegis  SM-3  Block 

IB  Missiles 

Met 

GAO  did  not  concur  with  one  of  the  three  criteria  for  MDA’s  rating 
because  the  costs  in  the  estimating  model  and  cost  review  file  were 
inconsistent. 

Aegis 

Modernization 

Met 

GAO  concurred  with  all  three  of  the  criteria  for  MDA’s  rating. 

Command,  Control, 
Battle  Management 
and 

Communications 
Spiral  8.2 

Partially  Met 

GAO  concurred  with  all  three  of  the  criteria  for  MDA’s  rating. 

Ground-based 
Midcourse  Defense 

Met 

GAO  concurred  with  all  three  of  the  criteria  for  MDA’s  rating. 

Targets  and 
Countermeasures 

Met 

GAO  concurred  with  all  three  of  the  criteria  for  MDA’s  rating. 

Terminal  High 
Altitude  Area 

Defense 

Met 

GAO  concurred  with  all  three  of  the  criteria  for  MDA’s  rating. 

AN/TPY-2 

Substantially  Met 

GAO  did  not  concur  with  two  of  the  three  criteria  for  MDA’s  rating 
because  documents  did  not  include  a  discussion  of  or  an  estimate  of 
disposal  costs. 

Upgraded  Early 
Warning  Radar 

Met 

GAO  concurred  with  all  three  of  the  criteria  for  MDA’s  rating. 

Facilities® 

Met 

GAO  concurred  with  all  three  of  the  criteria  for  MDA’s  rating. 

Global 

Deployment® 

Met 

GAO  concurred  with  all  three  of  the  criteria  for  MDA’s  rating. 

Source:  GAO  analysis  of  MDA  data.  |  GAO-1 5-21  OR 

®  MDA  provided  cost  estimates  for  Facilities  and  Global  Deployment  which  are  portions  of  the  Aegis  Ashore  cost 
estimate  as  presented  in  the  2014  baseline  report. 


MDA’s  Independent  Cost  Assessment  Team  Could  Help  to  Improve  the  Quality  of 
Estimates  but  It  Is  Too  Early  to  Assess 

Commensurate  with  a  GAO  best  practice,  MDA  also  assembled  an  independent  cost 
assessment  team  to  obtain  an  additional  review  of  the  quality  of  its  cost  estimates;  however,  the 
team  has  not  yet  defined  its  processes  or  produced  enough  results  for  us  to  assess.  According 
to  GAO’s  guide,  an  independent  cost  assessment  team  may  be  used  to  determine  whether  the 
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cost  estimate  reflects  the  program  of  record  by  providing  an  evaluation  of  a  cost  estimate’s 
quality  and  accuracy,  looking  specifically  at  a  program’s  technical  approach,  risk,  and 
acquisition  strategy  to  ensure  that  the  program’s  cost  estimate  captures  all  requirements.  MDA’s 
independent  cost  assessment  team,  established  in  January  2013,  collects  MDA’s  cost  estimates 
and  supporting  documentation,  and  then  uses  the  MDA  handbook  as  its  reference  point  to 
evaluate  how  well  the  estimates  are  meeting  the  standards  established  in  the  handbook.  The 
results  of  the  independent  group’s  assessment  are  included  in  a  detailed  report,  with 
descriptions  for  why  the  estimates  met  or  failed  to  meet  each  criteria  and  subcriteria  related  to 
the  characteristics  of  a  high  quality  estimate  as  outlined  in  the  MDA  handbook.  At  the  time  of 
our  review,  the  independent  cost  team  had  only  produced  two  detailed  reports  on  its  findings 
related  to  one  program’s  2013  estimate.  According  to  the  team’s  Deputy  Director,  it  was  also 
currently  drafting  a  guide  detailing  the  team’s  procedures  and  reporting  format  after 
incorporating  lessons  learned  from  its  2013  reviews.  The  team  plans  to  finalize  its  guide  by 
January  2015  and  complete  a  review  for  all  of  the  program  estimates  in  the  annual  baseline 
report  by  the  end  of  2015. 

CAPE  Independent  Cost  Estimates  Validate  Some  MDA  Programs,  but  Costs  for  About 
Half  of  the  Programs  Remain  Unverified 

While  the  CAPE  has  conducted  independent  cost  assessments  for  MDA  programs  in  the  past 
few  years  and  its  estimates  have  provided  some  credibility  for  MDA’s  estimates,  about  half  of 
the  cost  baselines  that  MDA  presents  in  its  annual  baseline  report  to  Congress  still  remain 
unverified.  One  of  the  criteria  for  a  credible  cost  estimate  is  having  an  independent  cost 
estimate  conducted  by  an  organization  independent  of  the  acquisition  chain  of  command. 
According  to  GAO’s  guide,  an  independent  cost  estimate  is  based  on  the  same  detailed 
technical  and  procurement  information  used  to  make  the  program’s  estimate.  The  independent 
estimate  might  cover  a  program’s  entire  life  cycle,  one  program  phase,  or  one  high-value,  highly 
visible,  or  high-interest  item  within  a  phase.  Because  the  team  performing  the  independent  cost 
estimate  is  independent,  it  provides  an  unbiased  test  of  whether  the  program  office  cost 
estimate  is  reasonable.  It  is  also  used  to  identify  risks  related  to  budget  shortfalls  or  excesses. 

Other  DOD  major  defense  acquisition  programs  are  required  to  have  an  independent  cost 
estimate  prepared  or  approved  by  the  CAPE  which  consists  of  full  life-cycle  cost — all  costs  of 
development,  procurement,  military  construction,  and  operations  and  support,  without  regard  to 
funding  source  or  management  control — before  advancing  through  major  milestones.®  Because 
of  the  flexibilities  granted  to  MDA  in  2002,  elements  of  the  BMDS  are  not  yet  required  to  obtain 
independent  cost  estimates,  and  while  it  has  not  required  its  programs  to  do  so,  the  agency  has 
requested  some  independent  cost  estimates  from  the  CAPE.''® 

In  2010,  we  recommended  that  MDA  obtain  CAPE  independent  cost  estimates  for  its  program 
baselines.''''  MDA  concurred  with  our  recommendation  and  has  since  received  CAPE 
independent  cost  assessments  and  estimates  for  three  of  the  nine  MDA  programs  presented  in 

0  U.S.C.  §  2434  requires  an  independent  cost  estimate  of  the  full  life-cycle  cost  of  the  program  before  a  major 
defense  acquisition  program  can  advance  into  system  development  and  demonstration  (now  known  as  engineering 
and  manufacturing  development)  or  production  and  deployment. 

'“Because  the  BMDS  has  not  yet  formally  entered  the  DOD  acquisition  cycle,  the  requirement  at  10  U.S.C.  §  2434, 
which  is  triggered  by  the  phases  of  the  cycle,  does  not  yet  apply. 

"GAO,  Defense  Acquisitions:  Missiie  Defense  Transition  Provides  Opportunity  to  Strengthen  Acquisition  Approach, 
GAO-1 0-311  (Washington,  D.C.:  Feb.  25,  2010). 
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the  2014  baseline  report.''^  The  CAPE  is  also  currently  finalizing  its  results  for  an  additional 
independent  cost  assessment  for  a  portion  of  the  costs  of  missiles  for  a  fourth  MDA  program, 
the  Terminal  High  Altitude  Area  Defense.  In  addition,  CAPE  memos  and  briefings  from  2012 
and  2014,  which  present  comparisons  of  its  results  to  MDA’s  estimates,  show  that  the  estimates 
are  close  and  appear  to  provide  validation  for  many  areas  of  MDA’s  estimates.  According  to  the 
GAO  cost  guide,  independent  cost  estimates  for  DOD  programs  are  usually  higher,  and  more 
accurate,  than  baseline  estimates.  Thus,  if  a  program  cost  estimate  is  close  to  independent  cost 
estimate  results,  one  can  be  more  confident  that  it  is  accurate  and  more  likely  to  result  in 
funding  at  a  reasonable  level. 

About  half  of  MDA  programs  still  remain  unverified.  Five  programs  or  program  components 
presenting  resource  baselines  in  the  2014  baseline  report  still  have  not  received  CAPE 
independent  cost  estimates.''^  By  the  time  the  CAPE  completes  its  assessment  for  the  Terminal 
High  Altitude  Area  Defense  missiles,  the  agency  will  have  received  independent  results  for 
about  half  of  its  programs,  representing  less  than  25  percent  of  the  total  baselined  costs  in 
MDA's  2014  baseline  report. 

Concluding  Observations 

MDA  has  made  a  considerable  effort  to  improve  its  cost  estimating  practices  since  our  initial 
review  in  201 1 ,  but  time  will  tell  as  MDA’s  internal  cost  review  and  its  independent  cost 
assessment  teams  continue  to  evaluate  how  well  the  agency  implements  recommendations 
from  prior  reviews  to  improve  future  estimates.  Our  assessment  of  MDA’s  cost  reviews 
indicates,  and  continued  evaluation  by  MDA’s  independent  cost  assessment  team  will  provide 
additional  assurance,  that  the  agency  is  able  to  identify  areas  where  the  cost  estimate  follows  its 
guide  and  where  correction  is  needed.  MDA’s  establishment  of  internal  cost  reviews  and  an 
independent  cost  assessment  team  are  best  practices  that  will  enable  the  agency  to  assess 
how  well  it  is  correcting  those  deficiencies  over  time  and  doing  the  real  work  of  improving  the 
quality  of  the  estimates.  Additionally,  MDA  has  made  progress  in  obtaining  independent 
estimates  from  the  CAPE  for  its  baselined  programs,  and  additional  estimates  from  the  CAPE 
will  further  work  to  validate  MDA  estimates  and  provide  more  credibility  to  MDA’s  cost 
estimating  results. 

Agency  Comments 


We  are  not  making  any  recommendations  in  this  report  because  we  have  previously 
recommended  that  the  Secretary  of  Defense  ensure  that  MDA: 

•  take  steps  to  ensure  cost  estimates  for  each  acquisition  baseline  are  high  quality, 
reliable,  and  documented  to  facilitate  external  review 

•  obtain  CAPE  independent  cost  estimates  for  each  baseline 


2012,  the  CAPE  completed  an  independent  cost  assessment  which  included  the  CAPE’s  estimates  for  Aegis 
Ashore,  Aegis  SM-3  Block  IB  and  associated  software,  and  Aegis  Modernization.  In  addition,  in  2014,  it  completed  an 
independent  cost  assessment  for  a  portion  of  the  costs  of  Aegis  SM-3  Block  IB  missiles. 

1 

These  five  programs  or  program  components  include  Command,  Control,  Battle  Management  and  Communications 
Spiral  8.2,  Ground-based  Midcourse  Defense,  Targets  and  Countermeasures,  Upgraded  Early  Warning  Radars,  and 
AN/TPY-2. 
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We  provided  a  draft  of  this  report  to  MDA  for  comment.  The  agency  provided  technical 
comments  which  were  incorporated  as  appropriate. 


We  are  sending  copies  of  this  report  to  the  appropriate  congressional  committees,  the  Secretary 
of  Defense,  and  to  the  Director,  MDA.  In  addition,  the  report  is  available  at  no  charge  on  the 
GAO  website  at  http://www.gao.gov. 


If  you  or  your  staff  have  any  questions  about  this  report,  please  contact  me  at  (202)  512-4841  or 
chaplainc@gao.gov.  Contact  points  for  our  Offices  of  Congressional  Relations  and  Public 
Affairs  may  be  found  on  the  last  page  of  this  report.  GAO  staff  who  made  key  contributions  to 
this  report  include  David  B.  Best,  Assistant  Director,  LaTonya  Miller,  Assistant  Director,  Brian 
Bothwell,  Meredith  Allen  Kimmett,  Joseph  W.  Kirschbaum,  Jennifer  Leotta,  Kenneth  E.  Patton, 
Karen  Richey,  and  Alyssa  Weir. 


Cristina  Chaplain 

Director,  Acquisition  and  Sourcing  Management 


Enclosure 
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List  of  Committees 


The  Honorable  Carl  Levin 
Chairman 

The  Honorable  James  Inhofe 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Richard  J.  Durbin 
Chairman 

The  Honorable  Thad  Cochran 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Howard  P.  “Buck”  McKeon 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Rodney  Frelinghuysen 
Chairman 

The  Honorable  Pete  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Enclosure:  GAO’s  Annual  Reports  on  Missile  Defense 


Report 

Number 

Report 

Date 

Report  Title 

GAO-04-409 

Apr.  2004 

Missile  Defense:  Actions  Are  Needed  to  Enhance  Testing  and 
Accountability 

GAO-05-243 

Mar.  2005 

Defense  Acquisitions:  Status  of  Ballistic  Missile  Defense  Program 
in  2004 

GAO-06-327 

Mar.  2006 

Defense  Acquisitions:  Missile  Defense  Agency  Fields  Initial 
Capability  but  Falls  Short  of  Original  Goals 

GAO-07-387 

Mar.  2007 

Defense  Acquisitions:  Missile  Defense  Acquisitions  Strategy 
Generates  Results  but  Delivers  Less  at  a  Higher  Cost 

GAO-08-448 

Mar.  2008 

Defense  Acquisitions:  Progress  Made  in  Fielding  Missile  Defense, 
but  Program  Is  Short  of  Meeting  Goals 

GAO-09-338 

Mar.  2009 

Defense  Acquisitions:  Production  and  Fielding  of  Missile  Defense 
Components  Continue  with  Less  Testing  and  Validation  Than 
Planned 

GAO-1 0-311 

Feb.  2010 

Defense  Acquisitions:  Missile  Defense  Transition  Provides 
Opportunity  to  Strengthen  Acquisition  Approach 

GAO-1 1-372 

Mar.  2011 

Missile  Defense:  Actions  Needed  to  Improve  Transparency  and 
Accountability 

GAO- 12-486 

Apr.  2012 

Missile  Defense:  Opportunity  Exists  to  Strengthen  Acquisitions  by 
Reducing  Concurrency 

GAO-13-432 

Apr.  2013 

Missile  Defense:  Opportunity  to  Refocus  on  Strengthening 
Acquisition  Management 

GAO-14-351 

Apr.  2014 

Missile  Defense:  Mixed  Progress  in  Achieving  Acquisition  Goals 
and  Improving  Accountability 

Source:  GAO.  |  GAO-15-210R 


Note:  Since  2002,  GAO  has  prepared  annual  assessments  of  MDA’s  progress  toward  its  acquisition  goals  according  to 
congressional  mandates  included  in  the  National  Defense  Authorization  Act  for  Fiscal  Year  2002,  Pub.  L.  No.  107-107,  §  232(g) 
(2001 );  Ronald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2005,  Pub.  L.  No.  1 08-375,  §  233  (2004);  National 
Defense  Authorization  Act  for  Fiscal  Year  2006,  Pub.  L.  No.  109-163,  §  232;  John  Warner  National  Defense  Authorization  Act  for 
Fiscal  Year  2007,  Pub.  L.  No.  1 09-364,  §  224  (2006);  National  Defense  Authorization  Act  for  Fiscal  Year  2008,  Pub.  L.  No.  110-181, 
§  225;  and  National  Defense  Authorization  Act  for  Fiscal  Year  2012,  Pub.  L.  No.  112-81 ,  §  232  (201 1 ). 


(121250) 
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This  is  a  work  of  the  U.S.  government  and  is  not  subject  to  copyright  protection  in  the 
United  States.  The  published  product  may  be  reproduced  and  distributed  in  its  entirety 
without  further  permission  from  GAO.  However,  because  this  work  may  contain 
copyrighted  images  or  other  material,  permission  from  the  copyright  holder  may  be 
necessary  if  you  wish  to  reproduce  this  material  separately. 


GAO’S  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  website  (http://www.gao.gov).  Each  weekday 
afternoon,  GAO  posts  on  its  website  newly  released  reports,  testimony, 
and  correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted 
products,  go  to  http://www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  website, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 
Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512- 
4400,  U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room 
7125,  Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngc1@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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